Skip to content

fix(lexer): end an operator where the next token starts - #21

Merged
aurelleb merged 2 commits into
vicinaehq:mainfrom
nkapila6:fix/lexer-leading-decimal
Oct 27, 2026
Merged

aurelleb merged 2 commits into
vicinaehq:mainfrom
nkapila6:fix/lexer-leading-decimal

Conversation

@nkapila6

Copy link
Copy Markdown
Contributor

Discovered this while daily driving Vicinae using Numen backend on the launcher too. FWIW, some of these do and don't happen on SoulverCore (the other backend option on Vicinae).

The Operator state only ended on a digit, ASCII letter or space, so anything else that starts a token got glued onto the operator. 3*.3 became 3 + "*." + 3. Same for $, quotes and UTF-8 (3*$5, 2*"a", 3*€2).

In the REPL 3*.3 silently returns 3. On the launcher you see no result at all, since Vicinae runs Numen in strict mode.

While lexing an operator, each next char either continues it (**, <=) or ends it:

3*(2)   '(' is an operator on its own   -> '*' ends   (#10)
3*.3    '.3' starts a number            -> '*' ends   (this PR)
3**2    '*' could make '**'             -> continue

#10 covered chars that are operators on their own. This covers chars that start a different token (number, word, string), i.e. anything Reset wouldn't route to Operator.

Testing

Added to tests/corpus/arithmetic.corpus, after the leading-zero section:

# an operator glued to a following leading-dot decimal reads the same as the spaced form
3*.3                        => 0.9
2**.5                       => 1.414214
2<.5                        => false
1!=.5                       => true

main vs this branch:

$ echo "3*.3" | ./main/build/numen
3*.3 = 3
$ echo "3*.3" | ./fix/build/numen
3*.3 = 0.9

$ echo "3*.1" | ./main/build/numen
3*.1 = 3
$ echo "3*.1" | ./fix/build/numen
3*.1 = 0.3

$ echo "3*0.1" | ./main/build/numen
3*0.1 = 0.3

Other cases on this branch:

❯ for e in '3*.3' '3*$5' '2*"a"' '2*π' '2<=3' '2*(3)'; printf '%s\n' $e | ./build/numen; end
3*.3 = 0.9
3*$5 = 55.09aed
Error: Invalid operands: Number and Text
Error: Expected EOF, looks like there is nothing we can parse!
2<=3 = true
2*(3) = 6

2*"a" and 2*π now error instead of silently returning 2; the spaced forms error the same way. aed is from my machine's region (UAE) via implicit currency conversion, same as 3 * $5.

numen-tests: all new assertions pass. 3 currency tests fail on both main and this branch with No conversion rate available for aed., for the same region reason. Unrelated to this change.

@nkapila6

nkapila6 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor Author

All tests pass when the locale is pinned to en_US:

❯ LC_ALL=en_US.UTF-8 ./build/numen-tests
Randomness seeded to: 4223859856
===============================================================================
All tests passed (14812 assertions in 93 test cases)

@aurelleb

Copy link
Copy Markdown
Contributor

thx

@aurelleb
aurelleb merged commit e837f69 into vicinaehq:main Oct 27, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants